Image processing apparatus, imaging apparatus, image processing method and program

ABSTRACT

Provided are an apparatus and a method for generating accurate distance information of a subject. The apparatus has an image processing unit which inputs a first image constituted by a visible light component and a second image including a visible light component and an infrared light component to calculate a subject distance, in which an image processing unit calculates two distance information of a TOF distance, which is the subject distance calculated according to a TOF system by utilizing the second image, and a stereo distance calculated according to a stereo system by utilizing the first image and the second image, determines TOF distance reliability indicating reliability of the TOF distance, and generates, as final distance information, the stereo distance, which is the subject distance according to the stereo system, or distance information calculated by synthesis processing of the TOF distance and the stereo distance, for a pixel region in which the reliability of the TOF distance is low.

TECHNICAL FIELD

The present disclosure relates to an image processing apparatus, animaging apparatus, an image processing method and a program. Moreparticularly, the present disclosure relates to an image processingapparatus, an imaging apparatus, an image processing method and aprogram for measuring a distance to a subject.

BACKGROUND ART

A time of flight (TOF) camera has been known as a camera which measuresa distance to a subject.

The TOF camera irradiates the subject with infrared light and calculatesthe distance from the time required for the reflected infrared light tobe incident on the camera.

Note that examples of the conventional technologies disclosed for theTOF system include Patent Document 1 (Japanese Patent ApplicationLaid-Open No. 2013-220254), Patent Document 2 (Japanese PatentApplication Laid-Open No. 2016-006627), and the like.

However, such a distance measuring system utilizing the infrared lighthas a problem that it is difficult to measure the distance, for example,outdoors where the sunlight is strong, and for a far subject which theirradiation infrared light does not reach.

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2013-220254-   Patent Document 2: Japanese Patent Application Laid-Open No.    2016-006627

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

The present disclosure has been made, for example, in light of the aboveproblems, and an object of one example of the present disclosure is toprovide an image processing apparatus, an imaging apparatus, an imageprocessing method and a program for enabling accurate distancemeasurement even in a case where accurate distance measurement by theTOF system is difficult.

Moreover, an object of one example of the present disclosure is toprovide an image processing apparatus, an imaging apparatus, an imageprocessing method and a program for generating an image with high imagequality to which a plurality of images are applied.

Solutions to Problems

According to a first aspect of the present disclosure,

an image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance,

in which the image processing unit includes:

a time of flight (TOF) system distance calculation unit which calculatesa TOF distance, which is the subject distance according to a TOF system,by utilizing an infrared light component of the second image;

a stereo system distance calculation unit which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

a TOF distance reliability determination unit which determinesreliability of the TOF distance; and

a subject distance information generation unit which generates, as finaldistance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

Moreover, according to a second aspect of the present disclosure,

an imaging apparatus includes:

a first imaging unit which captures a first image constituted by avisible light component;

a second imaging unit which captures a second image including a visiblelight component and an infrared light component; and

an image processing unit which inputs the first image and the secondimage and generates distance information which indicates a subjectdistance,

in which the image processing unit includes:

a time of flight (TOF) system distance calculation unit which executessubject distance calculation according to a TOF system by utilizing thesecond image;

a stereo system distance calculation unit which executes subjectdistance calculation according to a stereo system by utilizing the firstimage and the second image;

a TOF distance reliability determination unit which determinesreliability of a TOF distance which is the subject distance calculatedby the TOF system distance calculation unit; and

a subject distance information generation unit which generates finaldistance information on the basis of the reliability of the TOFdistance, and

the subject distance information generation unit generates, as the finaldistance information, the stereo distance, which is the subject distanceaccording to the stereo system, or the distance information calculatedby synthesis processing of the TOF distance and the stereo distance, fora pixel region in which the reliability of the TOF distance is low.

Furthermore, according to a third aspect of the present disclosure,

an image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

in which the first image is an image constituted by a visible lightcomponent,

the second image is an image including a visible light component and aninfrared light component, and

the image processing unit includes:

an infrared light separation unit which separates the second image intoa visible light component image and an infrared light component image;and

an image synthesis unit which executes synthesis processing of the firstimage and the visible light component image generated on the basis ofthe second image by the infrared light separation unit.

Further, a fourth aspect of the present disclosure is an imageprocessing method executed in an image processing apparatus,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance, and

the image processing unit executes:

time of flight (TOF) system distance calculation processing whichcalculates a TOF distance, which is the subject distance according to aTOF system, by utilizing an infrared light component of the secondimage;

stereo system distance calculation processing which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

TOF distance reliability determination processing which determinesreliability of the TOF distance; and

subject distance information generation processing which generates, asfinal distance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

Still further, a fifth aspect of the present disclosure is an imageprocessing method executed in an image processing apparatus,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

the first image is an image constituted by a visible light component,

the second image is an image including a visible light component and aninfrared light component, and

the image processing unit executes:

infrared light separation processing which separates the second imageinto a visible light component image and an infrared light componentimage; and

synthesis processing of the first image and the visible light componentimage generated on the basis of the second image.

Moreover, a sixth aspect of the present disclosure is a program forcausing an image processing apparatus to execute image processing,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance, and

the program causes the image processing unit to execute:

time of flight (TOF) system distance calculation processing whichcalculates a TOF distance, which is the subject distance according to aTOF system, by utilizing an infrared light component of the secondimage;

stereo system distance calculation processing which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

TOF distance reliability determination processing which determinesreliability of the TOF distance; and

subject distance information generation processing which generates, asfinal distance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

Furthermore, a seventh aspect of the present disclosure is a program forcausing an image processing apparatus to execute image processing,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

the first image is an image constituted by a visible light component,

the second image is an image including a visible light component and aninfrared light component, and

the program causes the image processing unit to execute:

infrared light separation processing which separates the second imageinto a visible light component image and an infrared light componentimage; and

synthesis processing of the first image and the visible light componentimage generated on the basis of the second image.

Note that the program of the present disclosure is a program which isprovided in a computer readable format and can be provided by a storagemedium or a communication medium to, for example, an informationprocessing apparatus or a computer system, which can execute variousprogram codes. By providing such a program in a computer readableformat, processings according to the program are realized on theinformation processing apparatus or the computer system.

Still other objects, features and advantages of the present disclosurewill become apparent from a more detailed description based on theexamples of the present disclosure described later and the accompanyingdrawings. Note that the term “system” in this specification refers to alogical group configuration of a plurality of apparatuses and is notlimited to a system in which the apparatus of each configuration is inthe same housing.

Effects of the Invention

According to the configuration of one example of the present disclosure,an apparatus and a method for generating accurate distance informationof a subject are realized.

Specifically, the apparatus has an image processing unit which inputs afirst image constituted by a visible light component and a second imageincluding a visible light component and an infrared light component tocalculate a subject distance, in which the image processing unitcalculates two distance information of a TOF distance, which is thesubject distance calculated according to a TOF system by utilizing thesecond image, and a stereo distance calculated according to a stereosystem by utilizing the first image and the second image, determines TOFdistance reliability indicating reliability of the TOF distance, andgenerates, as final distance information, the stereo distance, which isthe subject distance according to the stereo system, or distanceinformation calculated by synthesis processing of the TOF distance andthe stereo distance, for a pixel region in which the reliability of theTOF distance is low.

By these processings, the apparatus and the method for generating theaccurate distance information of the subject are realized.

Note that the effects described in this specification are merelyexamples and are not limited, and other additional effects may beprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration example of an imageprocessing apparatus.

FIG. 2 is a diagram illustrating the configuration and processings of animage processing unit.

FIG. 3 is a diagram illustrating infrared light separation processing.

FIG. 4 is a diagram illustrating the processing of a stereo distancereliability determination unit.

FIG. 5 is a diagram illustrating the processing of a TOF distancereliability determination unit.

FIG. 6 is a diagram illustrating one example of the processing executedby a subject distance information generation unit.

FIG. 7 is a diagram illustrating one example of the processing executedby the subject distance information generation unit.

FIG. 8 is a diagram illustrating one example of the processing executedby the subject distance information generation unit.

FIG. 9 is a diagram illustrating one example of the processing executedby the subject distance information generation unit.

FIG. 10 is a diagram showing a flowchart for explaining the distanceinformation generation processing sequence.

FIG. 11 is a diagram showing a flowchart for explaining the distanceinformation generation processing sequence.

FIG. 12 is a diagram showing a flowchart for explaining the distanceinformation generation processing sequence.

FIG. 13 is a diagram illustrating the configuration and processings ofthe image processing unit.

FIG. 14 is a diagram illustrating the configuration and processings ofan image synthesis unit.

FIG. 15 is a diagram illustrating the processing executed by a blendingexecution unit.

FIG. 16 is a diagram for explaining the effects of blending processing.

FIG. 17 is a diagram showing a flowchart for explaining the syntheticimage generation processing sequence.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, details of an image processing apparatus, an imagingapparatus, an image processing method and a program of the presentdisclosure will be described with reference to the drawings. Note thatthe description is given according to the following items.

1. About Configuration and Processings of Image Processing Apparatus ofPresent Disclosure

2. About Configuration and Processings of Image Processing Unit WhichGenerates Distance (Depth) Information

3. About Distance Information Calculation Processing Sequence Executedby Image Processing Apparatus

3-1. About Processing Sequence Using Two Reliability Information ofStereo Distance Reliability and TOF Distance Reliability

3-2. About Processing Sequence Using One Reliability Information of OnlyTOF Distance Reliability

3-3. About Processing Sequence for Selecting One of Stereo DistanceInformation and TOF Distance Information in Pixel Unit as Final DistanceInformation by Using One Reliability Information of Only TOF DistanceReliability

4. About Configuration and Processings of Image Processing Unit WhichGenerates Synthetic Image with Improved Image Quality

5. About Synthetic Image Generation Processing Sequence Executed byImage Processing Apparatus

6. Summary of Configurations of Present Disclosure

[1. About Configuration and Processings of Image Processing Apparatus ofPresent Disclosure]

The configuration and processings of the image processing apparatus ofthe present disclosure will be described with reference to FIG. 1 andthe followings.

FIG. 1 is a block diagram showing the configuration of an imagingapparatus, which is one example of an image processing apparatus 100 ofthe present disclosure.

Note that the image processing apparatus of the present disclosure isnot limited to the imaging apparatus, but also includes, for example, aninformation processing apparatus, such as a PC, which inputs a capturedimage of the imaging apparatus and executes image processing.

In the following, the configuration and processings of the imagingapparatus will be described as one example of the image processingapparatus 100 of the present disclosure.

The image processings other than the capturing processing described inthe following examples can be executed not only by the imagingapparatus, but also by an information processing apparatus such as a PC.

The image processing apparatus 100 as the imaging apparatus shown inFIG. 1 has a control unit 101, a storage unit 102, a codec 103, an inputunit 104, an output unit 105, an imaging unit 106 and an imageprocessing unit 120.

The imaging unit 106 has a first imaging unit 107 which performs onlynormal image capturing, and a second imaging unit 108 which performsinfrared light irradiation and performs capturing of an image includinginfrared light and visible light.

The first imaging unit 107 has a first imaging element 111 forperforming normal image capturing. The first imaging element 111 is, forexample, an RGB pixel array type imaging element which has an RGB colorfilter constituted by a Bayer array and outputs a signal correspondingto input light of each color of R, G and B in each pixel unit.Alternatively, the first imaging element may be a white and black (WB)sensor type imaging element which captures a monochrome image.

The second imaging unit 108 has an infrared light (IR) irradiation unit113 which outputs infrared light, and a second imaging element 112.

The second imaging unit 108 has the infrared light (IR) irradiation unit113 for measuring a subject distance by a time of flight (TOF) system,and the second imaging element 112 which receives infrared light andvisible light.

The time of flight (TOF) system is a system which irradiates the subjectwith the infrared light and calculates the subject distance from thetime taken for the reflected infrared light to be incident on thecamera.

Note that the visible light region received by the second imagingelement 112 is preferably similar to a region of the first imagingelement 111. For example, in a case where the first imaging element 111is an RGB pixel array type imaging element, the second imaging element112 is also an RGB pixel array type imaging element. In a case where thefirst imaging element 111 is a white and black (WB) sensor type imagingelement, the second imaging element 112 is also a white and black (WB)sensor type imaging element.

However, the second imaging element 112 receives the visible lighttogether with the infrared light (IR), and the sensor output includes avisible light component and an infrared light (IR) component.

The first imaging unit 107 and the second imaging unit 108 are twoimaging units set at positions apart by a predetermined interval, andthe respective captured images are images from different viewpoints.

The same subject image is not captured on the corresponding pixels, thatis, the pixels at the same positions of the two images from thedifferent viewpoints, and a subject shift according to a disparityoccurs.

By utilizing this positional shift, the subject distance calculation bya stereo system is performed.

In a case where the captured image is a still image, the first imagingunit 107 and the second imaging unit 108 capture two still images at thesame timing. In a case of capturing a moving image, the captured frameof each imaging unit is a synchronized captured frame, that is, acontinuous image frame captured sequentially at the same timing.

Note that the control of these capturing timings is performed by thecontrol unit 101.

The control unit 101 controls various processings executed in theimaging apparatus 100, such as image capturing, signal processing on acaptured image, image recording processing, and display processing. Thecontrol unit 101 includes, for example, a CPU which executes processingsaccording to various processing programs stored in the storage unit 102,and the like, and functions as a data processing unit which executes theprograms.

The storage unit 102 is configured with a storage unit for capturedimages, further with a storage unit for the processing programs executedin the control unit 101 and various parameters, and still further with aRAM, a ROM and the like which function as working areas at the time ofthe data processing.

The codec 103 executes encoding and decoding processings such ascompression and decompression processings of the captured images.

The input unit 104 is, for example, a user manipulation unit, and inputscontrol information such as start, end, and various mode settings forcapturing.

The output unit 105 is configured with a display unit, a speaker and thelike, and is utilized to display the captured images, a through imageand the like, output sound, and the like.

The image processing unit 120 inputs the two images inputted from theimaging unit 106, applies these two images and calculates the subjectdistance (depth). Moreover, by synthesizing the two images, an imagewith high image quality in which noise is reduced is generated.

The image processing unit 120 outputs a generated image 151 and distance(depth) information 152.

These data are stored in, for example, the storage unit 102.Alternatively, the image 151 is outputted to the display unitconfiguring the output unit 105.

Furthermore, the distance (depth) information 152 is utilized forvarious processings executed in the control unit 102.

[2. About Configuration and Processings of Image Processing Unit whichGenerates Distance (Depth) Information]

Next, the configuration and processings of the image processing unit 120of the image processing apparatus 100 shown in FIG. 1 will be describedwith reference to FIG. 2 and the followings.

As previously mentioned, the image processing unit 120 inputs the twoimages inputted from the imaging unit 106, applies these two images andgenerates the distance (depth) information 152 indicating the subjectdistance (depth). Moreover, by synthesizing the two images, the image151 as the image with high image quality in which noise is reduced isgenerated.

First, the generation processing of the distance (depth) information 152executed in the image processing unit 120 will be described.

FIG. 2 is a block diagram showing the partial configuration of the imageprocessing unit 120 of the image processing apparatus 100.

FIG. 2 shows a configuration applied to the generation processing of thedistance (depth) information 152 among the configuration of the imageprocessing unit 120.

As shown in FIG. 2, the image processing unit 120 has an infrared light(IR) separation unit 121, a stereo system distance calculation unit 122,a TOF system distance calculation unit 123, a stereo distancereliability determination unit 124, a TOF distance reliabilitydetermination unit 125 and a subject distance information generationunit 126.

The image processing unit 120 outputs the distance (depth) information152 generated by the subject distance information generation unit 126.

The distance (depth) information 152 is data having distance informationin each pixel unit for the subject included in the captured images.

The input signal into the image processing unit 120 is each of thefollowing signals.

(1) A visible light image 200 inputted from the first imaging unit 107,and

(2) a visible light+infrared light image 201 inputted from the secondimaging unit 108.

First, the infrared light (IR) separation unit 121 inputs the visiblelight+infrared light image 201 inputted from the second imaging unit 108and executes infrared light (IR) separation processing on the visiblelight+infrared light image 201.

A specific example of the infrared light (IR) separation processingexecuted by the infrared light (IR) separation unit 121 will bedescribed with reference to FIG. 3.

FIG. 3 is a diagram illustrating each of the infrared light separationprocessings in a case where the second imaging element 112 of the secondimaging unit 108 has one of the following two configurations.

(1) Utility example of white and black (WB) sensor without IR cut filter

(2) Utility example of RGB sensor without IR cut filter

First, with reference to “(1) Utility example of white and black (WB)sensor without IR cut filter” in FIG. 3, the infrared light separationprocessing will be described in a case where the second imaging element112 of the second imaging unit 108 is a white and black (WB) sensorwithout an IR cut filter.

In this case, the infrared light (IR) separation unit 121 performs thefollowing processings on the output signal from the second imagingelement 112 of the second imaging unit 108 to separate the visible lightand the infrared light.

Infrared Light(IR)=Acquire From Black(B)Pixels

Visible Light=White(W)Pixel Output−Black(B)Pixel Output

However, it is preferable for the visible light that the average values(Ave) of the white (W) pixel output and the black (B) pixel output arecalculated in a pixel region unit of a predetermined region unit forphase matching, and the difference between the average values arecalculated as a visible light output signal. That is, the visible lightimage output is obtained according to the following expression.

Visible Light Image=Ave(White(W)Pixel Output)−Ave(Black(B)Pixel Output)

Next, with reference to “(2) Utility Example of RGB Sensor Without IRCut Filter” shown in FIG. 3, the infrared light separation processingwill be described in a case where the second imaging element 112 of thesecond imaging unit 108 is an RGB sensor without an IR cut filter.

In this case, the infrared light (IR) separation unit 121 executesmatrix operation shown in the following (Expression 1) on the outputsignal from the second imaging element 112 of the second imaging unit108 to separate the visible light and the infrared light.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{11mu} 1} \right\rbrack \mspace{526mu}} & \; \\{\begin{bmatrix}R \\G \\B \\{IR}\end{bmatrix} = {\begin{bmatrix}\alpha_{00} & \alpha_{01} & \alpha_{02} \\\alpha_{10} & \alpha_{11} & \alpha_{12} \\\alpha_{20} & \alpha_{21} & \alpha_{22} \\\alpha_{30} & \alpha_{31} & \alpha_{32}\end{bmatrix}\begin{bmatrix}R \\G \\B\end{bmatrix}}} & \left( {{Expression}\mspace{14mu} 1} \right)\end{matrix}$

In the above (Expression 1) , α₁₁ to α₃₂ are separation parametersdecided according to sensor characteristics.

Thus, depending on whether the second imaging element 112 of the secondimaging unit 108 is

(1) a white and black (WB) sensor without an IR cut filter, or

(2) an RGB sensor without an IR cut filter,

the infrared light (IR) separation unit 121 executes differentprocessings described with reference to FIG. 3 to separate the visiblelight and the infrared light from the output signal, of the secondimaging element 112 of the second imaging unit 108, that is, the“visible light+infrared light image 201” shown in FIG. 2.

As shown in FIG. 2, a visible light image 202 generated by theseparation processing of the infrared light (IR) separation unit 121 isinputted into the stereo system distance calculation unit 122.

Furthermore, an infrared light image 203 generated by the separationprocessing of the infrared light (IR) separation unit 121 is inputtedinto a TOF system distance calculation unit 123.

Next, the processing of the stereo system distance calculation unit 122will be described.

The stereo system distance calculation unit 122 inputs the followingimages.

(1) The visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108.

As previously mentioned, the first imaging unit 107 and the secondimaging unit 108 are two imaging units set at positions apart by apredetermined interval, and the respective captured images (the visiblelight image 200 and the visible light image 202) are images fromdifferent viewpoints.

The same subject image is not captured on the corresponding pixels, thatis, the pixels at the same positions of the two images from thedifferent viewpoints, that is, the visible light image 200 and thevisible light image 202, and a subject shift according to a disparityoccurs.

The stereo system distance calculation unit 122 utilizes this positionalshift to execute the subject distance calculation by the stereo system.

Specifically, first, the disparity amount is calculated by using twoimage signals of the visible light image 200 inputted from the firstimaging unit 107 and the visible light image 202 inputted from thesecond imaging unit 108. Moreover, the distance to the subject iscalculated by triangulation on the basis of the baseline length, whichis the interval between the first imaging unit 107 and the secondimaging unit 108, and the disparity amount.

Note that this distance calculation is executed in pixel unitsconstituting the image or pixel region units including a plurality ofpixels.

Subject distance information generated by the stereo system distancecalculation unit 122 is inputted as stereo distance information 204 intothe subject distance information generation unit 126 as shown in FIG. 2.

Next, the processing of the TOF system distance calculation unit 123will be described.

The TOF system distance calculation unit 123 inputs the following image.

(1) The infrared light (IR) image 203 generated from the captured imageof the second imaging unit 108.

As previously mentioned, the time of flight (TOF) system is a systemwhich irradiates the subject with the infrared light and calculates thesubject distance from the time taken for the reflected infrared light tobe incident on the camera.

The TOF system distance calculation unit 123 measures the time from theinfrared light irradiation timing of the infrared light (IR) irradiationunit 113 of the second imaging unit 108 to the infrared light receptiontiming of the second imaging element 112 and calculates the subjectdistance.

Note that this subject distance calculation is also executed in pixelunits or pixel region units including a predetermined number of pixels,similarly to the stereo system previously mentioned.

However, such a distance measuring system utilizing the infrared lighthas a problem that it is difficult to measure the distance, for example,outdoors where the sunlight is strong, and for a far subject which theirradiation infrared light does not reach.

Subject distance information generated by the TOF system distancecalculation unit 123 is inputted as TOF distance information 205 intothe subject distance information generation unit 126 as shown in FIG. 2.

Next, the processing executed by the stereo distance reliabilitydetermination unit 124 will be described.

The stereo distance reliability determination unit 124 determineswhether or not the subject distance information generated by the stereosystem distance calculation unit 122 is reliable data, generates stereoreliability 206 including the determination information, and outputs thestereo reliability 206 to the subject distance information generationunit 126 as shown in FIG. 2.

Note that the stereo reliability 206 generated by the stereo distancereliability determination unit 124 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the stereo system distance calculation unit 122.

A specific example of the reliability determination processing executedby the stereo distance reliability determination unit 124 will bedescribed with reference to FIG. 4.

The example shown in FIG. 4 is processing of determining the reliabilityby using variance values of block configuration pixels applied to blockmatching processing in detection of the corresponding points of the twoimages executed in the stereo system distance calculation unit 122.

In the stereo system distance calculation unit 122, for the imagescaptured from two different viewpoints, that is,

(1) the visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108,

so-called block matching processing of detecting corresponding pixelblocks between these images, that is, pixel blocks assumed to havecaptured the same subject, is executed.

In this block matching, when a characteristic image such as an edge anda texture is included in the utilized pixel block, matching(association) can be correctly performed. That is, highly precise blockmatching becomes possible, and highly precise distance calculationbecomes possible. On the other hand, it is difficult to perform correctmatching (association) for, for example, a flat image region without acharacteristic, such as sky. As a result, highly precise distancecalculation becomes difficult.

The example shown in FIG. 4 is an example of the reliabilitydetermination processing of the stereo distance utilizing thischaracteristic.

In the graph shown in FIG. 4, the horizontal axis is the variance valueof the block configuration pixel applied to the block matchingprocessing, and the vertical axis is the reliability β of the stereodistance.

Note that the reliability β of the stereo distance is set in the rangefrom zero to one, and the lower the numerical value the lower thereliability, the higher the numerical value the higher the reliability.

A case where the variance value of the block is large means that manycharacteristic images, for example, images of edge portions, texturesand the like are included in the block, which means that this block is acharacteristic block which enhances the precision of the block matching.

In such a case where the variance value of the block is large, thereliability β of the stereo distance calculated by the stereo systemdistance calculation unit 122 is a higher value, that is, a value closeto one.

On the other hand, a case where the variance value of the block is smallmeans that the block has a few images of the edge portions, textures andthe like and is constituted by a flat image with a small change in thepixel value, for example, of sky or the like, which means this block isa block which lowers the precision of the block matching.

In such a case where the variance value of the block is small, thereliability β of the stereo distance calculated by the stereo systemdistance calculation unit 122 is a lower value, that is, a value closeto zero.

The stereo distance reliability determination unit 124 executes thereliability β of the stereo distance calculated by the stereo systemdistance calculation unit 122, for example, in block units and generatesthe distance information reliability in block units or blockconfiguration pixel units.

This reliability information is the stereo reliability 206 shown in FIG.2.

The stereo distance reliability determination unit 124 outputs thegenerated stereo reliability 206 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

Next, the processing executed by the TOF distance reliabilitydetermination unit 125 will be described.

The TOF distance reliability determination unit 125 determines whetheror not the subject distance information generated by the TOF systemdistance calculation unit 123 is reliable data, generates TOFreliability 207 including the determination information, and outputs theTOF reliability 207 to the subject distance information generation unit126 as shown in FIG. 2.

Note that the TOF reliability 206 generated by the TOF distancereliability determination unit 125 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the TOF system distance calculation unit 123.

A specific example of the reliability determination processing executedby the TOF distance reliability determination unit 125 will be describedwith reference to FIG. 5.

The example shown in FIG. 5 is processing of determining the reliabilityby using the amount of the received light at a time of non-irradiationof the infrared light (IR) utilized for the distance measurementaccording to the TOF system executed in the TOF system distancecalculation unit 123.

As previously mentioned, in the TOF system distance calculation unit123, the time from the infrared light irradiation timing of the infraredlight (IR) irradiation unit 113 of the second imaging unit 108 to theinfrared light reception timing of the second imaging element 112 ismeasured, and the subject distance is calculated.

However, infrared light also exists in nature, and sunlight inparticular includes many infrared light components.

The second imaging element 112 of the second imaging unit 108 receivesnot only the infrared light by the irradiation of the infrared light(IR) irradiation unit 113, but also such infrared light other than theirradiation light of the infrared light (IR) irradiation unit 113.

For example, in a case where an image is captured under sunlightincluding infrared light components, such as outdoors on a sunny day,the second imaging element 112 receives a lot of the infrared light innature other than the infrared light by the irradiation of the infraredlight (IR) irradiation unit 113. In such a situation, the measurementprecision of the time from the infrared light irradiation timing of theinfrared light (IR) irradiation unit 113 to the infrared light receptiontiming of the second imaging element 112 lowers. As a result, highlyprecise distance calculation becomes difficult.

On the other hand, for example, in a case where an image is captured inan environment, such as at night or indoors, where there is littleinfluence of sunlight including infrared light components, thepossibility that second imaging element 112 receives the infrared lightother than the illumination light of the infrared light (IR) irradiationunit 113 is reduced. As a result, the measurement precision of the timefrom the infrared light irradiation timing of the infrared light (IR)irradiation unit 113 to the infrared light reception timing of thesecond imaging element 112 is enhanced, enabling highly precise distancecalculation.

The example shown in FIG. 5 is an example of the reliabilitydetermination processing of the TOF distance utilizing thischaracteristic.

In the graph shown in FIG. 5, the horizontal axis is the received lightintensity of the infrared light (IR) by the second imaging element 112at a time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113, and the vertical axis is the reliability α ofthe TOF distance.

Note that the reliability α of the TOF distance is set in the range fromzero to one, and the lower the numerical value the lower thereliability, the higher the numerical value the higher the reliability.

A case where the received light intensity of the infrared light is largeat a time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there are a lot of infrared lightof external factors such as sunlight, which means that it is difficultto measure the TOF distance accurately.

In such a case where the received light intensity is large at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a lower value, thatis, a value close to zero.

On the other hand, a case where the received light intensity is small atthe time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there is a little infrared light ofexternal factors such as sunlight, which means that it is possible tomeasure the TOF distance accurately.

In such a case where the received light intensity is small at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a higher value, thatis, a value close to one.

The TOF distance reliability determination unit 125 calculates thereliability α of the TOF distance calculated by the TOF system distancecalculation unit 123, for example, in pixel units or pixel region units.

This reliability information is the TOF reliability 207 shown in FIG. 2.

The TOF distance reliability determination unit 125 outputs thegenerated TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

Next, the subject distance information generation processing executed bythe subject distance information generation unit 126 will be described.

As shown in FIG. 2, the subject distance information generation unit 126inputs each of the following data.

(1) The stereo distance information 204 calculated by the stereo systemdistance calculation unit 122,

(2) the TOF distance information 205 calculated by the TOF systemdistance calculation unit 123,

(3) the stereo reliability 206 generated by the stereo distancereliability determination unit 124, and

(4) the TOF reliability 207 generated by the TOF distance reliabilitydetermination unit 125.

The subject distance information generation unit 126 inputs each ofthese data, generates the final distance information which is one of thestereo distance information 204 calculated by the stereo system distancecalculation unit 122 and the TOF distance information 205 calculated bythe TOF system distance calculation unit 123 or is generated by blendingprocessing, and outputs the final distance information as the distance(depth) information 152.

Note that, on the basis of the stereo reliability 206 and the TOFreliability 207, the subject distance information generation unit 126generates the final distance information which is one of the distanceinformation determined to have high reliability or is generated by theblending processing, and outputs the final distance information as thedistance (depth) information 152.

Note that the generation of the final distance information based onthese reliability determinations is executed in pixel units or pixelregion units.

A specific processing example executed by the subject distanceinformation generation unit 126 will be described with reference to FIG.6 and the followings.

As described above, on the basis of the stereo reliability 206 and theTOF reliability 207, the subject distance information generation unit126 selects one of the distance information with high reliability orgenerates the final distance information by the blending processing, andoutputs the information as the distance (depth) information 152.

The example shown in FIG. 6 is a processing example in which the TOFdistance information 205 calculated by the TOF system distancecalculation unit 123 is set to be preferentially selected.

In the graph shown in FIG. 6,

the horizontal axis is the TOF reliability α generated by the TOFdistance reliability determination unit 125, and

the vertical axis is the stereo reliability β generated by the stereodistance reliability determination unit 124.

Both of the reliabilities α and β are values in the range from zero toone, the lowest reliability is zero, and the highest reliability is one.

The graph shown in FIG. 6 is divided into three regions of (a), (b) and(c).

The region (a) is a region meeting the following conditions of:

TOF reliability α≥Th1, and

Stereo reliability β=0 to 1.

Note that Th1 is a reliability threshold value, and, for example,Th1=0.5.

The region (b) is a region meeting the following conditions of:

TOF reliability α<Th1, and

Stereo reliability β≥Th2.

Note that Th2 is also a reliability threshold value, and, for example,Th2=0.5.

The region (c) is a region meeting the following conditions of:

TOF reliability α<Th1, and

Stereo reliability β<Th2.

The subject distance information generation unit 126 determines whichregion (a) to (c) that the two reliabilities,

(1) the stereo reliability β generated by the stereo distancereliability determination unit 124, and

(2) the TOF reliability α generated by the TOF distance reliabilitydetermination unit 125,

belong and generates the final distance information, that is, thedistance (depth) information 152, which is the output of the subjectdistance information generation unit 126 shown in FIG. 2, according toeach region as the following.

The region (a), that is, the region meeting the following conditions of:

TOF reliability α≥Th1, and

Stereo reliability β=0 to 1

is a region determined that the TOF reliability α is relatively high.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the TOF distance information205 calculated by the TOF system distance calculation unit 123 for thepixel or the pixel region corresponding to this region.

The region (b), that is, the region meeting the following conditions of:

TOF reliability α<Th1, and

Stereo reliability β≥Th2

is a region determined that the TOF reliability α is relatively low andthe stereo reliability β is relatively high.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the stereo distance information204 calculated by the stereo system distance calculation unit 122 forthe pixel or the pixel region corresponding to this region.

The region (c), that is, the region meeting the following conditions of:

TOF reliability α<Th1, and

Stereo reliability β<Th2

is a region determined that the TOF reliability α is relatively low andthe stereo reliability β is also relatively low.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the blending (synthesizing)processing result of the TOF distance information 205 calculated by theTOF system distance calculation unit 123 and the stereo distanceinformation 204 calculated by the stereo system distance calculationunit 122 for the pixel or the pixel region corresponding to this region.

Note that a specific example of the blending (synthesizing) processingwill be described later.

The processing example shown in FIG. 6 is a processing example in whichthe TOF distance information 205 calculated by the TOF system distancecalculation unit 123 is set to be preferentially selected.

Next, with reference to FIG. 7, a processing example, in which thestereo distance information 204 calculated by the stereo system distancecalculation unit 122 is set to be preferentially selected, will bedescribed.

Like the graph shown in FIG. 6, in the graph shown in FIG. 7,

the horizontal axis is the TOF reliability α generated by the TOFdistance reliability determination unit 125, and

the vertical axis is the stereo reliability β generated by the stereodistance reliability determination unit 124.

Both of the reliabilities α and β are values in the range from zero toone, the lowest reliability is zero, and the highest reliability is one.

The graph shown in FIG. 7 is divided into three regions of (d) , (e) and(f).

The region (d) is a region meeting the following conditions of:

Stereo reliability β≥Th2, and

TOF reliability α=0 to 1.

Note that Th2 is a reliability threshold value, and, for example,Th2=0.5.

The region (e) is a region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α≥Th1.

Note that Th1 is also a reliability threshold value, and, for example,Th1=0.5.

The region (f) is a region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α<Th1.

The subject distance information generation unit 126 determines whichregion (d) to (f) that the two reliabilities,

(1) the stereo reliability β generated by the stereo distancereliability determination unit 124, and

(2) the TOF reliability α generated by the TOF distance reliabilitydetermination unit 125,

belong and generates the final distance information, that is, thedistance (depth) information 152, which is the output of the subjectdistance information generation unit 126 shown in FIG. 2, according toeach region as the following.

The region (d), that is, the region meeting the following conditions of:

Stereo reliability β≥Th2, and

TOF reliability α=0 to 1

is a region determined that the stereo reliability β is relatively high.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the stereo distance information204 calculated by the stereo system distance calculation unit 122 forthe pixel or the pixel region corresponding to this region.

The region (e), that is, the region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α≥Th1

is a region determined that the stereo reliability β is relatively lowand the TOF reliability α is relatively high.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the TOF distance information205 calculated by the TOF system distance calculation unit 123 for thepixel or the pixel region corresponding to this region.

The region (f), that is, the region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α<Th1

is a region determined that the stereo reliability β is relatively lowand the TOF reliability α is also relatively low.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the blending (synthesizing)processing result of the TOF distance information 205 calculated by theTOF system distance calculation unit 123 and the stereo distanceinformation 204 calculated by the stereo system distance calculationunit 122 for the pixel or the pixel region corresponding to this region.

Note that a specific example of the blending (synthesizing) processingwill be described later.

Moreover, with reference to FIG. 8, still another processing example, inwhich the stereo distance information 204 calculated by the stereosystem distance calculation unit 122 is set to be preferentiallyselected, will be described.

Like the graph shown in FIG. 6, in the graph shown in FIG. 8,

the horizontal axis is the TOF reliability α generated by the TOFdistance reliability determination unit 125, and

the vertical axis is the stereo reliability β generated by the stereodistance reliability determination unit 124.

Both of the reliabilities α and β are values in the range from zero toone, the lowest reliability is zero, and the highest reliability is one.

The graph shown in FIG. 8 is divided into two regions of (g) and (h).

The region (g) is a region meeting one of the following conditions of:

Stereo reliability β≥Th2, and

TOF reliability α=0 to 1,

and,

Stereo reliability β<Th2, and

TOF reliability α<Th1.

Note that Th1 and Th2 are reliability threshold values, and, forexample, Th1=0.5 and Th2=0.5.

The region (h) is a region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α≥Th1.

The subject distance information generation unit 126 determines whichregion (g) or (h) that the two reliabilities,

(1) the stereo reliability β generated by the stereo distancereliability determination unit 124, and

(2) the TOF reliability α generated by the TOF distance reliabilitydetermination unit 125,

belong and generates the final distance information, that is, thedistance (depth) information 152, which is the output of the subjectdistance information generation unit 126 shown in FIG. 2, according toeach region as the following.

The region (g), that is, the region meeting one of the followingconditions of:

Stereo reliability β≥Th2, and

TOF reliability α=0 to 1,

and,

Stereo reliability β<Th2, and

TOF reliability α<Th1,

is one of regions of a region in which the stereo reliability β isrelatively high and a region in which both the stereo reliability β andthe TOF reliability α are relatively low.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the stereo distance information204 calculated by the stereo system distance calculation unit 122 forthe pixel or the pixel region corresponding to this region.

The region (h), that is, the region meeting the following conditions of:

Stereo reliability β<Th2, and

TOF reliability α≥Th1

is a region determined that the stereo reliability β is relatively lowand the TOF reliability α is relatively high.

The subject distance information generation unit 126 sets, as the finaldistance information, that is, the configuration data of the distance(depth) information 152 shown in FIG. 2, the blending (synthesizing)processing result of the TOF distance information 205 calculated by theTOF system distance calculation unit 123 and the stereo distanceinformation 204 calculated by the stereo system distance calculationunit 122 for the pixel or the pixel region corresponding to this region.

Note that a specific example of the blending (synthesizing) processingwill be described later.

As described with reference to FIGS. 6, 7 and 8, the subject distanceinformation generation unit 126 determines which predefined reliabilitysection region that the two reliabilities,

(1) the stereo reliability β generated by the stereo distancereliability determination unit 124, and

(2) the TOF reliability α generated by the TOF distance reliabilitydetermination unit 125,

belong and generates the final distance information, that is, thedistance (depth) information 152, which is the output of the subjectdistance information generation unit 126 shown in FIG. 2, according toeach region as the following.

A specific example of the blending (synthesizing) processing of the twodistance information executed by the subject distance informationgeneration unit 126 will be described.

FIG. 9 shows a processing example of the subject distance informationgeneration unit 126 similar to the processing example described withreference to FIG. 8.

FIG. 9(1) shows a processing example of a case where the TOF distancereliability is estimated to be relatively high (Th1≤TOF reliability α).

This corresponds to the right half region (Th1≤TOF reliability α) of thegraph in FIG. 8.

FIG. 9(h) corresponds to the region in FIG. 8(h), performs the blendingprocessing of the stereo distance information and the TOF distanceinformation, and sets the blending (synthesizing) processing result asthe final distance information.

FIG. 9(g 1) corresponds to the right side region (Th1≤TOF reliability α)in FIG. 8(g)

In this region, since the stereo reliability β of the stereo distanceinformation is sufficiently high, the stereo distance information is setas the final distance information.

FIG. 9(2) shows a processing example of a case where the TOF distancereliability is estimated to be relatively low (TOF reliability α<Th1).

This corresponds to the left half region (TOF reliability α<Th1) of thegraph in FIG. 8.

FIG. 9(g 2) corresponds to the left side region (TOF reliability α<Th1)in FIG. 8(g).

In this region, since the TOF reliability α of the TOF distanceinformation is low, the stereo distance information is set as the finaldistance information.

FIG. 9(1) shows a specific processing example of the blending processingof the stereo distance information and the TOF distance information.

Various processings are possible for the blending processing of thestereo distance information and the TOF distance information.

The following three blending processing examples will be described.

(a) Blending processing by averaging

(b) Blending processing in which the TOF reliability α is applied as ablending ratio setting parameter

(c) Blending processing in which the stereo reliability β is applied asa blending ratio setting parameter

The final distance information [depth] by these three types of blendingprocessings is calculated by the following (Expression 2a) to(Expression 2c) when

the stereo distance information 204 generated by the stereo systemdistance calculation unit 122 is “depth_(stereo),” and

the TOF distance information 205 generated by the TOF system distancecalculation unit 123 is [depth_(TOF)].

$\begin{matrix}{\left\lbrack {{Math}.\mspace{11mu} 2} \right\rbrack \mspace{509mu}} & \; \\{{(a)\; {Average}\mspace{14mu} {value}}{{depth} = \frac{\left( {{depth}_{Stereo} + {depth}_{TOF}} \right.}{2}}} & \left( {{Expression}\mspace{14mu} 2a} \right) \\{{(b)\mspace{11mu} {Blending}\mspace{14mu} {according}\mspace{14mu} {to}{\mspace{14mu} \;}{TOF}\mspace{14mu} {reliability}\mspace{14mu} \alpha}{{depth} = {{\left( {1 - \alpha} \right) \times {depth}_{Stereo}} + {\alpha \times {depth}_{TOF}}}}} & \left( {{Expression}\mspace{14mu} 2b} \right) \\{{(c)\; {Blending}\mspace{14mu} {according}\mspace{14mu} {to}\mspace{14mu} {stereo}\mspace{14mu} {reliability}\mspace{14mu} \beta}{{depth} = {{\beta \times {depth}_{Stereo}} + {\left( {1 - \beta} \right) \times {depth}_{TOF}}}}} & \left( {{Expression}\mspace{14mu} 2c} \right)\end{matrix}$

Note that the above (Expression 2a) to (Expression 2c) are one exampleof the blending processing of the distance information, and the blendingprocessing according to other various aspects can be applied.

[3. About Distance Information Calculation Processing Sequence Executedby Image Processing Apparatus]

Next, the distance information calculation processing sequence executedby the image processing apparatus will be described with reference tothe flowcharts in FIG. 10 and the followings.

The flowcharts shown in FIGS. 10 to 12 are flowcharts for explainingthree different kinds of distance information calculation processingsequences executed by the image processing apparatus 100.

Specifically, the flowcharts correspond to the distance informationcalculation processing sequences of the following aspects, respectively.

(1) A processing sequence using two reliability information of thestereo distance reliability and the TOF distance reliability (FIG. 10)

(2) A processing sequence using one reliability information of only theTOF distance reliability (FIG. 11)

(3) A processing sequence for selecting one of the stereo distanceinformation and the TOF distance information in pixel units as the finaldistance information by using one reliability information of only theTOF distance reliability (FIG. 12)

Note that the flowcharts shown in FIG. 10 and the followings areexecuted, for example, under the control of the control unit (dataprocessing unit) including a CPU which executes processings according tothe processing programs stored in the storage unit, and the like.

Hereinafter, the processing in each step of the flowchart shown in FIG.10 will be sequentially described.

[3-1. About Processing Sequence Using Two Reliability Information ofStereo Distance Reliability and TOF Distance Reliability]

First, with reference to the flowchart shown in FIG. 10, the processingsequence using two reliability information of the stereo distancereliability and the TOF distance reliability will be described.

Hereinafter, the processing in each step will be sequentially described.

(Steps S101 a and S101 b)

Steps S101 a and S101 b are image capturing processings.

The two images are captured in the first imaging unit 107 and the secondimaging unit 108 shown in FIGS. 1 and 2.

Step S101 a is the capturing processing of the visible light image 200in the first imaging unit 107 shown in FIG. 2.

Step S101 b is the capturing processing of the visible light+infraredlight image 201 in the second imaging unit 108 shown in FIG. 2.

(Step S102)

Step S102 is the processing executed by the infrared light (IR)separation unit 121 shown in FIG. 2.

In Step S102, the infrared light (IR) separation unit 121 inputs thevisible light+infrared light image 201 captured by the second imagingunit 108 in Step S101 b, executes the infrared light (IR) separationprocessing, and generates the visible light image 202 and the infraredlight image 203 shown in FIG. 2.

This infrared light (IR) separation processing is the processingpreviously described with reference to FIG. 3.

(Step S103)

The processing in the next Step S103 is the processing executed by theTOF system distance calculation unit 123 shown in FIG. 2.

In Step S103, the TOF system distance calculation unit 123 executes thesubject distance calculation processing according to the time of flight(TOF) system.

The TOF system distance calculation unit 123 utilizes the infrared lightimage 203 generated by the infrared light (IR) separation unit 121 inStep S102 to measure the time from the infrared light irradiation timingof the infrared light (IR) irradiation unit 113 of the second imagingunit 108 shown in FIG. 2 to the infrared light reception timing of thesecond imaging element 112, and calculates the subject distance.

Note that this subject distance calculation is executed in pixel unitsor pixel region units including a predetermined number of pixels.

(Step S104)

The processing in the next Step S104 is the processing executed by thestereo system distance calculation unit 122 shown in FIG. 2.

In Step S104, the stereo system distance calculation unit 122 executesthe subject distance calculation processing according to the stereosystem.

Specifically, the distance to the subject is calculated by triangulationbased on the disparity amount calculated by using the two image signalsof the visible light image 200 captured by the first imaging unit 107 inStep S101 a and the visible light image 202 captured by the secondimaging unit 108 in Step S101 b and generated in Step S102, and thebaseline length which is the interval between the first imaging unit 107and the second imaging unit 108.

Note that this distance calculation is executed in pixel unitsconstituting the image or pixel region units including a plurality ofpixels.

(Step S105)

The processing in the next Step S105 is the processing executed by thestereo distance reliability determination unit 124 shown in FIG. 2.

In Step S105, the stereo distance reliability determination unit 124determines whether or not the subject distance information generated bythe stereo system distance calculation unit 122 is reliable data,generates the stereo reliability 206 including the determinationinformation, and outputs the stereo reliability 206 to the subjectdistance information generation unit 126 as shown in FIG. 2.

Note that the stereo reliability 206 generated by the stereo distancereliability determination unit 124 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the stereo system distance calculation unit 122.

As previously described with reference to FIG. 4, for example, thestereo distance reliability determination unit 124 determines thereliability by using the variance values of the block configurationpixels applied to the block matching processing in the stereo systemdistance calculation unit 122.

In a case where the block variance value is large, the stereo distancereliability β is a higher value, that is, a value close to one. On theother hand, in a case where the variance value of the block is small,the stereo distance reliability β is a lower value, that is, a valueclose to zero.

(Step S106)

The processing in the next Step S106 is the processing executed by theTOF distance reliability determination unit 125 shown in FIG. 2.

In Step S106, the TOF distance reliability determination unit 125determines whether or not the subject distance information generated bythe TOF system distance calculation unit 123 is reliable data, generatesthe TOF reliability 207 including the determination information, andoutputs the TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

Note that the TOF reliability 206 generated by the TOF distancereliability determination unit 125 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the TOF system distance calculation unit 123.

The reliability determination processing executed by the TOF distancereliability determination unit 125 is, for example, the processingpreviously described with reference to FIG. 5.

That is, the reliability is determined according to the input amount ofthe exogenous infrared light to the light receiving element at a time ofnon-irradiation of the infrared light by the infrared light (IR)irradiation unit 113.

A case where the received light intensity of the infrared light is largeat a time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there are a lot of infrared lightof external factors such as sunlight, which means that it is difficultto measure the TOF distance accurately.

In such a case where the received light intensity is large at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a lower value, thatis, a value close to zero.

On the other hand, a case where the received light intensity is small atthe time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there is a little infrared light ofexternal factors such as sunlight, which means that it is possible tomeasure the TOF distance accurately.

In such a case where the received light intensity is small at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a higher value, thatis, a value close to one.

The TOF distance reliability determination unit 125 calculates thereliability α of the TOF distance calculated by the TOF system distancecalculation unit 123, for example, in pixel units or pixel region units.

This reliability information is the TOF reliability 207 shown in FIG. 2.

The TOF distance reliability determination unit 125 outputs thegenerated TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

(Step S107)

The processing in Step S107 is the processing executed by the subjectdistance information generation unit 126 shown in FIG. 2.

On the basis of the stereo distance reliability 206 and the TOF distancereliability 207, the subject distance information generation unit 126confirms the reliabilities of the stereo distance information 204 andthe TOF distance information 205, selects one of the distanceinformation or generates the synthesizing result of the two distanceinformation, and generates the information or the result as the finaloutput distance information.

Note that this processing is executed in pixel units or pixel regionunits constituted by a predetermined number of pixels.

On the basis of the stereo reliability 206 and the TOF reliability 207,the subject distance information generation unit 126 selects any one ofthe distance information determined to have high reliability, orgenerates new distance information by the blending processing, andoutputs either one of them as the final distance information, that is,the distance (depth) information 152.

These specific processing examples are as described with reference toFIGS. 6 to 9.

(Step S108)

Next, in Step S108, it is determined whether or not the generation ofthe final distance information has been completed for all the pixels.

In a case where there is a pixel which has not been completed, theprocessing returns to Step S105, and the processings in Step S105 andthe followings are executed for the unprocessed pixel.

In Step S108, when it is determined that the generation of the finaldistance information has been completed for all the pixels, theprocessing ends.

At this point, the distance (depth) information 152 shown in FIG. 2 isoutputted from the image processing unit 120.

This distance (depth) information 152 is distance (depth) information inwhich one of the following distance information of

(a) the stereo distance information,

(b) the TOF distance information, and

(c) the synthetic distance information of the stereo distanceinformation and the TOF distance information,

is set in pixel units or pixel region units.

For the distance information associated with each pixel, distanceinformation with high reliability is selected, and highly precisedistance information is outputted for the entire image.

[3-2. About Processing Sequence Using One Reliability Information ofOnly TOF Distance Reliability]

Next, with reference to the flowchart shown in FIG. 11, the processingsequence using one reliability information of only the TOF distancereliability will be described.

Hereinafter, the processing in each step will be sequentially described.

(Steps S101 to S104)

The processings in Steps S101 to S104 are processings similar to theprocessings in Steps S101 to S104 previously described with reference tothe flowchart in FIG. 10.

Step S101 a is the capturing processing of the visible light image 200in the first imaging unit 107 shown in FIG. 2.

Step S101 b is the capturing processing of the visible light+infraredlight image 201 in the second imaging unit 108 shown in FIG. 2.

Step S102 is the processing executed by the infrared light (IR)separation unit 121 shown in FIG. 2, which inputs the visiblelight+infrared light image 201 captured by the second imaging unit 108,executes the infrared light (IR) separation processing, and generatesthe visible light image 202 and the infrared light image 203 shown inFIG. 2.

The processing in Step S103 is the subject distance calculationprocessing according to the time of flight (TOF) system executed by theTOF system distance calculation unit 123 shown in FIG. 2. The subjectdistance (TOF distance) is calculated by utilizing the infrared lightimage 203 generated by the infrared light (IR) separation unit 121.

The processing in Step S104 is the processing executed by the stereosystem distance calculation unit 122 shown in FIG. 2. The stereo systemdistance calculation unit 122 calculates the subject distance (stereodistance) by using the two image signals of the visible light image 200captured by the first imaging unit 107 and the visible light image 202obtained from the captured image of the second imaging unit 108.

Note that this distance calculation is executed in pixel unitsconstituting the image or pixel region units including a plurality ofpixels.

(Step S151)

The processing in the next Step S151 is the processing executed by theTOF distance reliability determination unit 125 shown in FIG. 2.

In Step S151, the TOF distance reliability determination unit 125determines whether or not the subject distance information generated bythe TOF system distance calculation unit 123 is reliable data, generatesthe TOF reliability 207 including the determination information, andoutputs the TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

Note that the TOF reliability 206 generated by the TOF distancereliability determination unit 125 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the TOF system distance calculation unit 123.

The reliability determination processing executed by the TOF distancereliability determination unit 125 is, for example, the processingpreviously described with reference to FIG. 5.

That is, the reliability is determined according to the input amount ofthe exogenous infrared light to the light receiving element at a time ofnon-irradiation of the infrared light by the infrared light (IR)irradiation unit 113.

A case where the received light intensity of the infrared light is largeat a time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there are a lot of infrared lightof external factors such as sunlight, which means that it is difficultto measure the TOF distance accurately.

In such a case where the received light intensity is large at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a lower value, thatis, a value close to zero.

On the other hand, a case where the received light intensity is small atthe time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there is a little infrared light ofexternal factors such as sunlight, which means that it is possible tomeasure the TOF distance accurately.

In such a case where the received light intensity is small at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a higher value, thatis, a value close to one.

The TOF distance reliability determination unit 125 calculates thereliability α of the TOF distance calculated by the TOF system distancecalculation unit 123, for example, in pixel units or pixel region units.

This reliability information is the TOF reliability 207 shown in FIG. 2.

The TOF distance reliability determination unit 125 outputs thegenerated TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

(Step S152)

The processing in Step S152 is the processing executed by the subjectdistance information generation unit 126 shown in FIG. 2.

On the basis of the TOF distance reliability 207, the subject distanceinformation generation unit 126 generates one of the following distanceinformation of

(a) the stereo distance information,

(b) the TOF distance information, and

(c) the synthetic distance information of the stereo distanceinformation and the TOF distance information

as the final output distance information.

Note that this processing is executed in pixel units or pixel regionunits constituted by a predetermined number of pixels.

In the present example, the stereo reliability 206 is not used, but theoutput information is generated on the basis of only the TOF reliability207 and outputted as the distance (depth) information 152.

(Step S153)

Next, in Step S153, it is determined whether or not the generation ofthe final distance information has been completed for all the pixels.

In a case where there is a pixel which has not been completed, theprocessing returns to Step S151, and the processings in Step S151 andthe followings are executed for the unprocessed pixel.

In Step S153, when it is determined that the generation of the finaldistance information has been completed for all the pixels, theprocessing ends.

At this point, the distance (depth) information 152 shown in FIG. 2 isoutputted from the image processing unit 120.

This distance (depth) information 152 is distance (depth) information inwhich one of the following distance information of

(a) the stereo distance information,

(b) the TOF distance information, and

(c) the synthetic distance information of the stereo distanceinformation and the TOF distance information,

is set in pixel units or pixel region units.

For the distance information associated with each pixel, distanceinformation with high reliability is selected, and highly precisedistance information is outputted for the entire image.

[3-3. About Processing Sequence for Selecting One of Stereo DistanceInformation and TOF Distance Information in Pixel Unit as Final DistanceInformation by Using One Reliability Information of Only TOF DistanceReliability]

Next, with reference to the flowchart shown in FIG. 12, the processingsequence for selecting one of the stereo distance information and theTOF distance information in pixel units as the final distanceinformation by using one reliability information of only the TOFdistance reliability will be described.

Hereinafter, the processing in each step will be sequentially described.

(Steps S101 to S104)

The processings in Steps S101 to S104 are processings similar to theprocessings in Steps S101 to S104 previously described with reference tothe flowchart in FIG. 10.

Step S101 a is the capturing processing of the visible light image 200in the first imaging unit 107 shown in FIG. 2.

Step S101 b is the capturing processing of the visible light+infraredlight image 201 in the second imaging unit 108 shown in FIG. 2.

Step S102 is the processing executed by the infrared light (IR)separation unit 121 shown in FIG. 2, which inputs the visiblelight+infrared light image 201 captured by the second imaging unit 108,executes the infrared light (IR) separation processing, and generatesthe visible light image 202 and the infrared light image 203 shown inFIG. 2.

The processing in Step S103 is the subject distance calculationprocessing according to the time of flight (TOF) system executed by theTOF system distance calculation unit 123 shown in FIG. 2. The subjectdistance (TOF distance) is calculated by utilizing the infrared lightimage 203 generated by the infrared light (IR) separation unit 121.

The processing in Step S104 is the processing executed by the stereosystem distance calculation unit 122 shown in FIG. 2. The stereo systemdistance calculation unit 122 calculates the subject distance (stereodistance) by using the two image signals of the visible light image 200captured by the first imaging unit 107 and the visible light image 202obtained from the captured image of the second imaging unit 108.

Note that this distance calculation is executed in pixel unitsconstituting the image or pixel region units including a plurality ofpixels.

(Step S181)

The processing in the next Step S181 is the processing executed by theTOF distance reliability determination unit 125 shown in FIG. 2.

In Step S181, the TOF distance reliability determination unit 125determines whether or not the subject distance information generated bythe TOF system distance calculation unit 123 is reliable data, generatesthe TOF reliability 207 including the determination information, andoutputs the TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

Note that the TOF reliability 206 generated by the TOF distancereliability determination unit 125 includes reliability information foreach of the subject distance information in pixel units or pixel regionunits generated by the TOF system distance calculation unit 123.

The reliability determination processing executed by the TOF distancereliability determination unit 125 is, for example, the processingpreviously described with reference to FIG. 5.

That is, the reliability is determined according to the input amount ofthe exogenous infrared light to the light receiving element at a time ofnon-irradiation of the infrared light by the infrared light (IR)irradiation unit 113.

A case where the received light intensity of the infrared light is largeat a time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there are a lot of infrared lightof external factors such as sunlight, which means that it is difficultto measure the TOF distance accurately.

In such a case where the received light intensity is large at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a lower value, thatis, a value close to zero.

On the other hand, a case where the received light intensity is small atthe time of non-irradiation of the infrared light by the infrared light(IR) irradiation unit 113 means that there is a little infrared light ofexternal factors such as sunlight, which means that it is possible tomeasure the TOF distance accurately.

In such a case where the received light intensity is small at the timeof non-irradiation of the infrared light by the infrared light (IR)irradiation unit 113, the reliability α of the TOF distance calculatedby the TOF system distance calculation unit 123 is a higher value, thatis, a value close to one.

The TOF distance reliability determination unit 125 calculates thereliability α of the TOF distance calculated by the TOF system distancecalculation unit 123, for example, in pixel units or pixel region units.

This reliability information is the TOF reliability 207 shown in FIG. 2.

The TOF distance reliability determination unit 125 outputs thegenerated TOF reliability 207 to the subject distance informationgeneration unit 126 as shown in FIG. 2.

(Step S182)

The processings in Steps S5182 to S184 are the processings executed bythe subject distance information generation unit 126 shown in FIG. 2.

On the basis of the TOF distance reliability 207, the subject distanceinformation generation unit 126 generates one of the following distanceinformation of

(a) the stereo distance information, and

(b) the TOF distance information,

as the final output distance information.

Note that this processing is executed in pixel units or pixel regionunits constituted by a predetermined number of pixels.

In the present example, the processing of synthesizing the stereodistance information and the TOF distance information is not executed,but one of the stereo distance information and the TOF distanceinformation is selected in pixel units as the final output distanceinformation.

In Step S182, it is determined whether or not the TOF reliability 207 islow, that is, the TOF reliability 207 is less than the predeterminedthreshold value. In a case where it is determined that the TOFreliability 207 is less than the predetermined threshold value and low,the processing proceeds to Step S183.

On the other hand, in a case where it is determined that the TOFreliability 207 is not low, that is, the TOF reliability 207 is equal toor greater than the predetermined threshold value and it is determinedthat the TOF reliability 207 is high, the processing proceeds to StepS184.

(Step S183)

In a case where it is determined in Step S182 that the TOF reliability207 is low, the subject distance information generation unit 126 selectsthe stereo distance information as the final output distance informationin Step S183.

(Step S184)

On the other hand, in a case where it is determined in Step S182 thatthe TOF reliability 207 is not low, the subject distance informationgeneration unit 126 selects the TOF distance information as the finaloutput distance information in Step S184.

(Step S185)

Next, in Step S185, it is determined whether or not the generation ofthe final distance information has been completed for all the pixels.

In a case where there is a pixel which has not been completed, theprocessing returns to Step S181, and the processings in Step S181 andthe followings are executed for the unprocessed pixel.

In Step S185, when it is determined that the generation of the finaldistance information has been completed for all the pixels, theprocessing ends.

At this point, the distance (depth) information 152 shown in FIG. 2 isoutputted from the image processing unit 120.

This distance (depth) information 152 is distance (depth) information inwhich one of the following distance information of

(a) the stereo distance information, and

(b) the TOF distance information,

is set in pixel units or pixel region units.

For the distance information associated with each pixel, distanceinformation with high reliability is selected, and highly precisedistance information is outputted for the entire image.

[4. About Configuration and Processings of Image Processing Unit WhichGenerates Synthetic Image with Improved Image Quality]

Next, with reference to FIG. 13 and the followings, the configurationand processings of the image processing unit, which generates asynthetic image with improved image quality, will be described.

As previously mentioned, the image processing unit 120 inputs the twoimages inputted from the imaging unit 106, applies these two images andgenerates the distance (depth) information 152 indicating the subjectdistance (depth) as well as generates the image 151 as an image withhigh image quality, in which noise is reduced, by synthesizing the twoimages.

Hereinafter, the generation processing of the synthetic image withimproved image quality in the image processing unit 120 will bedescribed.

FIG. 13 is a block diagram showing the partial configuration of theimage processing unit 120 of the image processing apparatus 100.

FIG. 13 shows a configuration applied to the generation processing of asynthetic image 410 among the configuration of the image processing unit120.

As shown in FIG. 13, the image processing unit 120 has the infraredlight (IR) separation unit 121 and an image synthesis unit 300.

The input signal into the image processing unit 120 is each of thefollowing signals.

(1) A visible light image 200 inputted from the first imaging unit 107,and

(2) a visible light+infrared light image 201 inputted from the secondimaging unit 108.

First, the infrared light (IR) separation unit 121 inputs the visiblelight+infrared light image 201 inputted from the second imaging unit 108and executes infrared light (IR) separation processing on the visiblelight+infrared light image 201.

The infrared light (IR) separation processing executed by the infraredlight (IR) separation unit 121 is the processing previously describedwith reference to FIG. 3.

The following images are inputted into the image synthesis unit 300.

(1) The visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108.

The configuration and processing example of the image synthesis unit 300will be described with reference to FIG. 14.

As shown in FIG. 14, the image synthesis unit 300 has an image shiftdetection unit 301, a blending ratio calculation unit 302 and a blendingexecution unit 303.

The image shift detection unit 301 inputs the following two images.

(1) The visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108.

The image shift detection unit 301 detects the positional shift of theimage for these two images. The positional shift amount in pixel unitsis calculated, and shift information 311 including shift amount data inpixel units is generated and outputted to the blending ratio calculationunit 302.

The blending ratio calculation unit 302 calculates the blending ratio ofthe pixels at the corresponding positions, that is, at the samecoordinate positions of the two images, that is, the following twoimages of

(1) the visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108,

on the basis of the “shift information 311” inputted from the imageshift detection unit 301, that is, the shift amount in pixel units.

Specifically, a high blending ratio is set for a pixel with a smallshift amount, and a small blending ratio is set for a pixel with a largeshift amount.

For example, the blending ratio is decided by the setting as shown inthe graph in FIG. 15.

In the graph shown in FIG. 15, the horizontal axis is the positionalshift amount of the corresponding pixels of the two images, and thevertical axis is the blending ratio.

Blending ratio=1 indicates that the pixels at the correspondingpositions of the two images are blended (synthesized) by 1:1. Blendingratio=0 means that pixel values of one image are directly outputtedwithout being blended.

Thus, the blending ratio calculation unit 302 calculates the blendingratio of the pixels at the corresponding positions, that is, at the samecoordinate positions of the two images on the basis of the “shiftinformation 311” inputted from the image shift detection unit 301, thatis, the shift amount in pixel units.

As shown in FIG. 14, the calculated blending ratio 312 is outputted tothe blending execution unit 303.

The blending execution unit 303 executes the blending processing of thepixels at the corresponding positions, that is, at the same coordinatepositions of two images on the basis of the “shift information 311”inputted from the blending ratio calculation unit 302, that is, theshift amount in pixel units, and generates and outputs an syntheticimage 410.

The synthetic image 410 becomes a high-quality image, in which noise isreduced, by synthesizing the two images.

Note that the level of image quality improvement expected by thissynthesis processing varies depending on the configuration of theimaging elements of the imaging units which capture images.

The correspondence between the configuration of the imaging elements andthe expected image quality improvement aspect will be described withreference to FIG. 16.

FIG. 16 shows the image quality improvement aspects realized by theabove synthesis processing of the two images in a case of the fourcombinations in which a case where the first imaging unit and the secondimaging unit are each the Bayer array, that is, the RGB pixel array, anda case where the first imaging unit and the second imaging unit are eachthe white array, that is, the WB pixel array.

In a case where both of the two imaging units are the Bayer arrays, thenoise reduction effect can be obtained for both signals of the luminancesignal and the chroma signal (color, chroma).

In addition, in a case where at least one of the imaging unit has animaging element of the white array, the noise reduction effect can beobtained for only the luminance signal.

[5. About Synthetic Image Generation Processing Sequence Executed byImage Processing Apparatus]

Next, with reference to a flowchart shown in FIG. 17, the generationprocessing sequence of the synthetic image with improved image qualityexecuted by the image processing apparatus will be described.

Hereinafter, the processing in each step will be sequentially described.

(Step S201)

Step S201 a is the capturing processing of the visible light image 200in the first imaging unit 107 shown in FIG. 2.

Step S201 b is the capturing processing of the visible light+infraredlight image 201 in the second imaging unit 108 shown in FIG. 2.

(Step S202)

Step S202 is the processing executed by the infrared light (IR)separation unit 121 shown in FIG. 2, which inputs the visiblelight+infrared light image 201 captured by the second imaging unit 108,executes the infrared light (IR) separation processing, and generatesthe visible light image 202 and the infrared light image 203 shown inFIG. 2.

As shown in FIGS. 13 and 14, the following images are inputted into theimage synthesis unit 300.

(1) The visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108.

(Step S203)

The processing in Step S203 is the processing executed by the imageshift detection unit 301 of the image synthesis unit 300 shown in FIG.14.

The image shift detection unit 301 inputs the following two images.

(1) The visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108.

The image shift detection unit 301 detects the positional shift of theimage for these two images. The positional shift amount in pixel unitsis calculated, and shift information 311 including shift amount data inpixel units is generated and outputted to the blending ratio calculationunit 302.

(Step S204)

The processing in Step S204 is the processing executed by the blendingratio calculation unit 302 of the image synthesis unit 300 shown in FIG.14.

The blending ratio calculation unit 302 calculates the blending ratio ofthe pixels at the corresponding positions, that is, at the samecoordinate positions of the two images, that is, the following twoimages of

(1) the visible light image 200 which is the captured image of the firstimaging unit 107, and

(2) the visible light image 202 generated from the captured image of thesecond imaging unit 108,

on the basis of the “shift information 311” inputted from the imageshift detection unit 301, that is, the shift amount in pixel units.

Specifically, a high blending ratio is set for a pixel with a smallshift amount, and a small blending ratio is set for a pixel with a largeshift amount. The calculated blending ratio 312 is outputted to theblending execution unit 303.

(Step S205)

The processing in Step S205 is the processing executed by the blendingexecution unit 303 of the image synthesis unit 300 shown in FIG. 14.

The blending execution unit 303 executes the blending processing of thepixels at the corresponding positions, that is, at the same coordinatepositions of two images on the basis of the “shift information 311”inputted from the blending ratio calculation unit 302, that is, theshift amount in pixel units, and calculates a correction pixel value ofeach pixel.

(Step S206)

Next, in Step S206, it is determined whether or not the correction pixelvalue calculation has been completed for all the pixels.

In a case where there is a pixel which has not been completed, theprocessing returns to Step S203, and the processings in Step S203 andthe followings are executed for the unprocessed pixel.

When it is determined in Step 206 that the correction pixel valuecalculation has been completed for all the pixels, the processingproceeds to Step S207.

(Step S207)

When the correction pixel value calculation has been completed for allthe pixels, the blending execution unit 303 of the image synthesis unit300 shown in FIG. 14 generates the synthetic image 410, in which thecorrection pixel values is set, to be outputted.

The synthetic image 410 becomes a high-quality image, in which noise isreduced, by synthesizing the two images.

[6. Summary of Configurations of Present Disclosure]

The examples of the present disclosure have been described in detailabove with reference to specific examples. However, it is obvious thatthose skilled in the art can make modifications and substitutions of theexamples in a scope without departing from the gist of the presentdisclosure. That is, the present invention has been disclosed in theform of exemplification and should not be interpreted restrictively. Inorder to judge the gist of the present disclosure, the scope of claimsshould be taken into consideration.

Note that the technology disclosed in this specification can adopt thefollowing configurations.

(1) An image processing apparatus including:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance,

in which the image processing unit includes:

a time of flight (TOF) system distance calculation unit which calculatesa TOF distance, which is the subject distance according to a TOF system,by utilizing an infrared light component of the second image;

a stereo system distance calculation unit which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

a TOF distance reliability determination unit which determinesreliability of the TOF distance; and

a subject distance information generation unit which generates, as finaldistance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

(2) The image processing apparatus according to (1), further including:

an infrared light separation unit which separates the second image intoa visible light component image and an infrared light component image,

in which the TOF system distance calculation unit executes subjectdistance calculation processing by utilizing the infrared lightcomponent image generated by the infrared light separation unit, and

the stereo system distance calculation unit executes subject distancecalculation processing by utilizing the visible light component imagegenerated by the infrared light separation unit.

(3) The image processing apparatus according to (1) or (2), in which theTOF distance reliability determination unit determines the reliabilityof the TOF distance according to an amount of an infrared lightcomponent included in the second image which is a captured image of thesecond imaging unit at a time of non-irradiation of infrared light.

(4) The image processing apparatus according to any one of (1) to (3),further including:

a stereo distance reliability determination unit which determinesreliability of the stereo distance which is the subject distancecalculated by the stereo system distance calculation unit,

in which the subject distance information generation unit generates, asthe final distance information, the TOF distance or distance informationcalculated by synthesis processing of the TOF distance and the stereodistance, for a pixel region in which the reliability of the stereodistance is low.

(5) The image processing apparatus according to (4), in which the stereodistance reliability determination unit determines the reliability ofthe stereo distance according to a variance value of a pixel value of ablock configuration pixel applied to block matching processing in thestereo system distance calculation unit.

(6) The image processing apparatus according to any one of (1) to (5),in which the subject distance information generation unit generates, asthe final distance information, one of following (a) to (c) distanceinformation of:

(a) the stereo distance,

(b) the TOF distance, and

(c) a synthetic distance of the stereo distance and the TOF distance,

in pixel unit or pixel region unit according to the reliability of theTOF distance in the pixel unit or the pixel region unit.

(7) The image processing apparatus according to any one of (1) to (6),in which the subject distance information generation unit generates, asthe final distance information, one of following (a) to (c) distanceinformation of:

(a) the stereo distance,

(b) the TOF distance, and

(c) the synthetic distance of the stereo distance and the TOF distance,

in the pixel unit or the pixel region unit according to the reliabilityof the stereo distance in the pixel unit or the pixel region unit.

(8) An imaging apparatus including:

a first imaging unit which captures a first image constituted by avisible light component;

a second imaging unit which captures a second image including a visiblelight component and an infrared light component; and

an image processing unit which inputs the first image and the secondimage and generates distance information which indicates a subjectdistance,

in which the image processing unit includes:

a time of flight (TOF) system distance calculation unit which executessubject distance calculation according to a TOF system by utilizing thesecond image;

a stereo system distance calculation unit which executes subjectdistance calculation according to a stereo system by utilizing the firstimage and the second image;

a TOF distance reliability determination unit which determinesreliability of a TOF distance which is the subject distance calculatedby the TOF system distance calculation unit; and

a subject distance information generation unit which generates finaldistance information on the basis of the reliability of the TOFdistance, and

the subject distance information generation unit generates, as the finaldistance information, the stereo distance, which is the subject distanceaccording to the stereo system, or the distance information calculatedby synthesis processing of the TOF distance and the stereo distance, fora pixel region in which the reliability of the TOF distance is low.

(9) An image processing apparatus including:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

in which the first image is an image constituted by a visible lightcomponent,

the second image is an image including a visible light component and aninfrared light component, and

the image processing unit includes:

an infrared light separation unit which separates the second image intoa visible light component image and an infrared light component image;and

an image synthesis unit which executes synthesis processing of the firstimage and the visible light component image generated on the basis ofthe second image by the infrared light separation unit.

(10) The image processing apparatus according to (9), in which the imagesynthesis unit includes:

an image shift calculation unit which calculates a positional shiftamount in pixel unit of the first image and the visible light componentimage generated on the basis of the second image by the infrared lightseparation unit;

a blending ratio calculation unit which calculates, according to thepositional shift amount calculated by the image shift calculation unit,a blending ratio in the pixel unit of the first image and the visiblelight component image generated on the basis of the second image by theinfrared light separation unit; and

a blending execution unit which executes, according to the blendingratio calculated by the blending ratio calculation unit, blendingprocessing in the pixel unit of the first image and the visible lightcomponent image generated on the basis of the second image by theinfrared light separation unit.

(11) The image processing apparatus according to (9) or (10), furtherincluding a time of flight (TOF) system distance calculation unit whichexecutes subject distance calculation according to a TOF system byutilizing the second image.

(12) The image processing apparatus according to any one of (9) to (11),further including a stereo system distance calculation unit whichexecutes subject distance calculation according to a stereo system byutilizing the first image and the second image.

(13) An image processing method executed in an image processingapparatus,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance, and

the image processing unit executes:

time of flight (TOF) system distance calculation processing whichcalculates a TOF distance, which is the subject distance according to aTOF system, by utilizing an infrared light component of the secondimage;

stereo system distance calculation processing which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

TOF distance reliability determination processing which determinesreliability of the TOF distance; and

subject distance information generation processing which generates, asfinal distance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

(14) An image processing method executed in an image processingapparatus,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

the first image is an image constituted by a visible light component,

the second image is an image including a visible light component and aninfrared light component, and

the image processing unit executes:

infrared light separation processing which separates the second imageinto a visible light component image and an infrared light componentimage; and

synthesis processing of the first image and the visible light componentimage generated on the basis of the second image.

(15) A program for causing an image processing apparatus to executeimage processing,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generatesdistance information which indicates a subject distance, and

the program causes the image processing unit to execute:

time of flight (TOF) system distance calculation processing whichcalculates a TOF distance, which is the subject distance according to aTOF system, by utilizing an infrared light component of the secondimage;

stereo system distance calculation processing which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage;

TOF distance reliability determination processing which determinesreliability of the TOF distance; and

subject distance information generation processing which generates, asfinal distance information, the stereo distance or synthetic distanceinformation of the TOF distance and the stereo distance, for a pixelregion in which the reliability of the TOF distance is low.

(16) A program for causing an image processing apparatus to executeimage processing,

in which the image processing apparatus includes:

an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image,

the first image is an image constituted by a visible light component,

the second image is an image including a visible light component and aninfrared light component, and

the program causes the image processing unit to execute:

infrared light separation processing which separates the second imageinto a visible light component image and an infrared light componentimage; and

synthesis processing of the first image and the visible light componentimage generated on the basis of the second image.

Moreover, the series of processings described in the specification canbe executed by hardware, software or a composite configuration thereof.In the case of executing the processings by software, it is possible toinstall a program, in which the processing sequences are recorded, in amemory inside a computer incorporated into dedicated hardware and causethe program to be executed or to install the program in ageneral-purpose computer, which can execute various processings, andcause the program to be executed. For example, the program can beprerecorded on a recording medium. Besides installing the program in thecomputer from the recording medium, it is possible to receive theprogram via a network such as local area network (LAN) and the Internetand install the program on a recording medium such as an incorporatedhard disk.

Note that the various processings described in the specification are notonly executed in time series according to the description but also maybe executed in parallel or individually according to the processingcapability of the apparatus which executes the processings or asnecessary. Furthermore, the term “system” in this specification refersto a logical group configuration of a plurality of apparatuses and isnot limited to a system in which the apparatus of each configuration isin the same housing.

INDUSTRIAL APPLICABILITY

As described above, according to the configuration of one example of thepresent disclosure, the apparatus and the method for generating accuratedistance information of a subject are realized.

Specifically, the apparatus has an image processing unit which inputs afirst image constituted by a visible light component and a second imageincluding a visible light component and an infrared light component tocalculate a subject distance, in which the image processing unitcalculates two distance information of a TOF distance, which is thesubject distance calculated according to a TOF system by utilizing thesecond image, and a stereo distance calculated according to a stereosystem by utilizing the first image and the second image, determines TOFdistance reliability indicating reliability of the TOF distance, andgenerates, as final distance information, the stereo distance, which isthe subject distance according to the stereo system, or distanceinformation calculated by synthesis processing of the TOF distance andthe stereo distance, for a pixel region in which the reliability of theTOF distance is low.

By these processings, the apparatus and the method for generating theaccurate distance information of the subject are realized.

REFERENCE SIGNS LIST

-   100 Image processing apparatus-   101 Control unit-   102 Storage unit-   103 Codec-   104 Input unit-   105 Output unit-   106 Imaging unit-   107 First imaging unit-   108 Second imaging unit-   111 First imaging element-   112 Second imaging element-   113 Infrared light (IR) irradiation unit-   120 Image processing unit-   121 Infrared light (IR) separation unit-   122 Stereo system distance calculation unit-   123 TOF system distance calculation unit-   124 Stereo distance reliability determination unit-   125 TOF distance reliability determination unit-   126 Subject distance information generation unit-   151 Image-   152 Distance (depth) information-   300 Image synthesis unit-   301 Image shift detection unit-   302 Blending ratio calculation unit-   303 Blending execution unit-   410 Synthetic image

1. An image processing apparatus comprising: an image processing unitwhich inputs a first image and a second image, which are captured imagesfrom two different viewpoints, and generates distance information whichindicates a subject distance, wherein the image processing unitcomprises: a time of flight (TOF) system distance calculation unit whichcalculates a TOF distance, which is the subject distance according to aTOF system, by utilizing an infrared light component of the secondimage; a stereo system distance calculation unit which calculates astereo distance, which is the subject distance according to a stereosystem, by utilizing the first image and a visible light component ofthe second image; a TOF distance reliability determination unit whichdetermines reliability of the TOF distance; and a subject distanceinformation generation unit which generates, as final distanceinformation, the stereo distance or synthetic distance information ofthe TOF distance and the stereo distance, for a pixel region in whichthe reliability of the TOF distance is low.
 2. The image processingapparatus according to claim 1, further comprising: an infrared lightseparation unit which separates the second image into a visible lightcomponent image and an infrared light component image, wherein the TOFsystem distance calculation unit executes subject distance calculationprocessing by utilizing the infrared light component image generated bythe infrared light separation unit, and the stereo system distancecalculation unit executes subject distance calculation processing byutilizing the visible light component image generated by the infraredlight separation unit.
 3. The image processing apparatus according toclaim 1, wherein the TOF distance reliability determination unitdetermines the reliability of the TOF distance according to an amount ofan infrared light component included in the second image which is acaptured image of the second imaging unit at a time of non-irradiationof infrared light.
 4. The image processing apparatus according to claim1, further comprising: a stereo distance reliability determination unitwhich determines reliability of the stereo distance which is the subjectdistance calculated by the stereo system distance calculation unit,wherein the subject distance information generation unit generates, asthe final distance information, the TOF distance or distance informationcalculated by synthesis processing of the TOF distance and the stereodistance, for a pixel region in which the reliability of the stereodistance is low.
 5. The image processing apparatus according to claim 4,wherein the stereo distance reliability determination unit determinesthe reliability of the stereo distance according to a variance value ofa pixel value of a block configuration pixel applied to block matchingprocessing in the stereo system distance calculation unit.
 6. The imageprocessing apparatus according to claim 1, wherein the subject distanceinformation generation unit generates, as the final distanceinformation, one of following (a) to (c) distance information of: (a)the stereo distance, (b) the TOF distance, and (c) a synthetic distanceof the stereo distance and the TOF distance, in pixel unit or pixelregion unit according to the reliability of the TOF distance in thepixel unit or the pixel region unit.
 7. The image processing apparatusaccording to claim 4, wherein the subject distance informationgeneration unit generates, as the final distance information, one offollowing (a) to (c) distance information of: (a) the stereo distance,(b) the TOF distance, and (c) a synthetic distance of the stereodistance and the TOF distance, in pixel unit or pixel region unitaccording to the reliability of the stereo distance in the pixel unit orthe pixel region unit.
 8. An imaging apparatus comprising: a firstimaging unit which captures a first image constituted by a visible lightcomponent; a second imaging unit which captures a second image includinga visible light component and an infrared light component; and an imageprocessing unit which inputs the first image and the second image andgenerates distance information which indicates a subject distance,wherein the image processing unit comprises: a time of flight (TOF)system distance calculation unit which executes subject distancecalculation according to a TOF system by utilizing the second image; astereo system distance calculation unit which executes subject distancecalculation according to a stereo system by utilizing the first imageand the second image; a TOF distance reliability determination unitwhich determines reliability of a TOF distance which is the subjectdistance calculated by the TOF system distance calculation unit; and asubject distance information generation unit which generates finaldistance information on the basis of the reliability of the TOFdistance, and the subject distance information generation unitgenerates, as the final distance information, the stereo distance, whichis the subject distance according to the stereo system, or the distanceinformation calculated by synthesis processing of the TOF distance andthe stereo distance, for a pixel region in which the reliability of theTOF distance is low.
 9. An image processing apparatus comprising: animage processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image, wherein the first image is an image constituted by avisible light component, the second image is an image including avisible light component and an infrared light component, and the imageprocessing unit comprises: an infrared light separation unit whichseparates the second image into a visible light component image and aninfrared light component image; and an image synthesis unit whichexecutes synthesis processing of the first image and the visible lightcomponent image generated on the basis of the second image by theinfrared light separation unit.
 10. The image processing apparatusaccording to claim 9, wherein the image synthesis unit comprises: animage shift calculation unit which calculates a positional shift amountin pixel unit of the first image and the visible light component imagegenerated on the basis of the second image by the infrared lightseparation unit; a blending ratio calculation unit which calculates,according to the positional shift amount calculated by the image shiftcalculation unit, a blending ratio in the pixel unit of the first imageand the visible light component image generated on the basis of thesecond image by the infrared light separation unit; and a blendingexecution unit which executes, according to the blending ratiocalculated by the blending ratio calculation unit, blending processingin the pixel unit of the first image and the visible light componentimage generated on the basis of the second image by the infrared lightseparation unit.
 11. The image processing apparatus according to claim9, further comprising a time of flight (TOF) system distance calculationunit which executes subject distance calculation according to a TOFsystem by utilizing the second image.
 12. The image processing apparatusaccording to claim 9, further comprising a stereo system distancecalculation unit which executes subject distance calculation accordingto a stereo system by utilizing the first image and the second image.13. An image processing method executed in an image processingapparatus, wherein the image processing apparatus comprises: an imageprocessing unit which inputs a first image and a second image, which arecaptured images from two different viewpoints, and generates distanceinformation which indicates a subject distance, and the image processingunit executes: time of flight (TOF) system distance calculationprocessing which calculates a TOF distance, which is the subjectdistance according to a TOF system, by utilizing an infrared lightcomponent of the second image; stereo system distance calculationprocessing which calculates a stereo distance, which is the subjectdistance according to a stereo system, by utilizing the first image anda visible light component of the second image; TOF distance reliabilitydetermination processing which determines reliability of the TOFdistance; and subject distance information generation processing whichgenerates, as final distance information, the stereo distance orsynthetic distance information of the TOF distance and the stereodistance, for a pixel region in which the reliability of the TOFdistance is low.
 14. An image processing method executed in an imageprocessing apparatus, wherein the image processing apparatus comprises:an image processing unit which inputs a first image and a second image,which are captured images from two different viewpoints, and generates asynthetic image, the first image is an image constituted by a visiblelight component, the second image is an image including a visible lightcomponent and an infrared light component, and the image processing unitexecutes: infrared light separation processing which separates thesecond image into a visible light component image and an infrared lightcomponent image; and synthesis processing of the first image and thevisible light component image generated on the basis of the secondimage.
 15. A program for causing an image processing apparatus toexecute image processing, wherein the image processing apparatuscomprises: an image processing unit which inputs a first image and asecond image, which are captured images from two different viewpoints,and generates distance information which indicates a subject distance,and the program causes the image processing unit to execute: time offlight (TOF) system distance calculation processing which calculates aTOF distance, which is the subject distance according to a TOF system,by utilizing an infrared light component of the second image; stereosystem distance calculation processing which calculates a stereodistance, which is the subject distance according to a stereo system, byutilizing the first image and a visible light component of the secondimage; TOF distance reliability determination processing whichdetermines reliability of the TOF distance; and subject distanceinformation generation processing which generates, as final distanceinformation, the stereo distance or synthetic distance information ofthe TOF distance and the stereo distance, for a pixel region in whichthe reliability of the TOF distance is low.
 16. A program for causing animage processing apparatus to execute image processing, wherein theimage processing apparatus comprises: an image processing unit whichinputs a first image and a second image, which are captured images fromtwo different viewpoints, and generates a synthetic image, the firstimage is an image constituted by a visible light component, the secondimage is an image including a visible light component and an infraredlight component, and the program causes the image processing unit toexecute: infrared light separation processing which separates the secondimage into a visible light component image and an infrared lightcomponent image; and synthesis processing of the first image and thevisible light component image generated on the basis of the secondimage.